Knowledge-based automatic image segmentation

ABSTRACT

A method for medical image segmentation. The method includes accessing and updating a knowledge-base in accordance with embodiments of the present invention. The techniques include: receiving a medical image and computing a sparse landmark signature based on the medical image content. Next, a knowledge-base is searched for representative matches to form a base set, wherein the base set comprises a plurality of reference image sets. A portion of the plurality of reference image sets of the base set is deformed to generate mappings from the base set to the medical image set. Finally a weighted average segmentation for each structure of interest of the medical image set is determined.

RELATED U.S. APPLICATION

This application is a continuation application of Ser. No. 12/845,358,entitled “Knowledge-Based Automatic Image Segmentation,” by C.Zankowski, filed Jul. 28, 2010, now U.S. Pat. No. 9,454,823, herebyincorporated by reference in entirety.

TECHNICAL FIELD

The present disclosure relates generally to the field of automatic imagesegmentation and more specifically to the field of medical imageautomatic segmentation configured for medical applications.

BACKGROUND

When medical imaging is necessary to observe an internal organ or a setof internal organs, there are several systems that may be utilized:X-ray, magnetic resonance imaging (MM), computed tomography (CT), andothers. When CT or MRI imagery, for example, is used, a series oftwo-dimensional images are taken from a three-dimensional volume. Here,each two-dimensional image is an image of a cross-sectional “slice” ofthe three-dimensional volume. The resulting collection oftwo-dimensional cross-sectional slices can be combined to create a threedimensional image or reconstruction of the patient's anatomy. Thisresulting three-dimensional image or three-dimensional reconstructionwill contain the desired internal organ. This portion of thethree-dimensional image or reconstruction that contains the structure ofinterest may be referred to as a volume of interest. Note that when itis desired to observe multiple internal organs, there will then be aplurality of structures of interest as well.

These one or more structures of interest may be viewed in several ways.A first and simplest way to view the structure(s) of interest would beto merely view the original CT or MRI image slices for the patient, witheach slice containing a view of the structure(s) of interest. A second,and more complicated method to view the structure(s) of interest wouldbe to combine the series of two-dimensional cross-sectional slices intoa single three-dimensional representation where the structure(s) ofinterest may be represented as solid, opaque, or translucent, etc.,objects that may then be manipulated (e.g., rotated) to allow viewingfrom multiple angles.

One purpose of the three-dimensional reconstruction of the structure(s)of interest containing diseased or abnormal tissues or organs is thepreparation of a three-dimensional radiation therapy treatment plan.Radiation therapy treatment plans are used during medical proceduresthat selectively expose precise areas of the body, such as canceroustumors, to specific doses of radiation to destroy the undesirabletissues. To develop a patient-specific radiation therapy treatment plan,information is extracted from the three-dimensional model to determineparameters such as organ shape, organ volume, tumor shape, tumorlocation in the organ, and the position or orientation of several otherstructures of interest as they relate to the affected organ and anytumor.

The two-dimensional slices may be individually viewed on a computerscreen and with the use of conventional graphics programs, the contoursof organs or structures of interest can be traced out by hand. Contoursare connected line segments that define the outline of a structure ofinterest, which may be an organ, a portion of an organ, a tumor,diseased tissue, or a whole patient outline. Alternatively, thesestructures of interest in specific organs such as the brain or prostate,for example, may be identified with various structure-specific automaticcontouring and/or automatic segmentation software programs (subdividingan image into discrete regions) that outline or fill the shape of thestructure of interest on each two-dimensional slice of a set of slices.

SUMMARY OF THE INVENTION

This present invention provides a solution to the challenges inherent inmedical image automatic segmentation. In a method according to oneembodiment, a series of steps provide knowledge-based medical imageautomatic segmentation. After receiving a medical image, aknowledge-base is searched for representative matches. A plurality ofreference image sets are selected to form a base set. A portion of thereference image sets in the base set are deformed to generate mappingsfrom the reference images to the medical image. A weighted averagesegmentation for each organ of interest is calculated. In oneembodiment, the weighted average segmentation for each organ of interestis used as a seed for an automated structure-specific segmentationalgorithm. In another embodiment, after the automatic segmentationalgorithms are complete, reviewed and corrected as necessary, themedical image and its associated meta-data, sparse landmark signaturesand structures of interest are added to the knowledge-base.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be better understood from a reading of thefollowing detailed description, taken in conjunction with theaccompanying drawing figures in which like reference charactersdesignate like elements and in which:

FIG. 1 is a simplified block diagram illustrating a system for providingknowledge-based medical image automatic segmentation, in accordance withan embodiment of the present invention;

FIG. 2 illustrates steps of an example process for knowledge-basedmedical image automatic segmentation, in accordance with an embodimentof the present invention;

FIG. 3 illustrates steps of an example process for knowledge-basedmedical image automatic segmentation, in accordance with an embodimentof the present invention;

FIG. 4 illustrates steps of an example process for knowledge-basedmedical image automatic segmentation of a cohort of reference image setsin the knowledge-base, in accordance with an embodiment of the presentinvention; and

FIG. 5 illustrates the steps of a process for selecting, deforming, andcomparing cohorts in accordance with an embodiment of the presentinvention.

DETAILED DESCRIPTION

Reference will now be made in detail to the preferred embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings. While the invention will be described in conjunction with thepreferred embodiments, it will be understood that they are not intendedto limit the invention to these embodiments. On the contrary, theinvention is intended to cover alternatives, modifications andequivalents, which may be included within the spirit and scope of theinvention as defined by the appended claims. Furthermore, in thefollowing detailed description of embodiments of the present invention,numerous specific details are set forth in order to provide a thoroughunderstanding of the present invention. However, it will be recognizedby one of ordinary skill in the art that the present invention may bepracticed without these specific details. In other instances, well-knownmethods, procedures, components, and circuits have not been described indetail so as not to unnecessarily obscure aspects of the embodiments ofthe present invention.

NOTATION AND NOMENCLATURE

Some portions of the detailed descriptions, which follow, are presentedin terms of procedures, steps, logic blocks, processing, and othersymbolic representations of operations on data bits within a computermemory. These descriptions and representations are the means used bythose skilled in the data processing arts to most effectively convey thesubstance of their work to others skilled in the art. A procedure,computer executed step, logic block, process, etc., is here, andgenerally, conceived to be a self-consistent sequence of steps orinstructions leading to a desired result. The steps are those requiringphysical manipulations of physical quantities. Usually, though notnecessarily, these quantities take the form of electrical or magneticsignals capable of being stored, transferred, combined, compared, andotherwise manipulated in a computer system. It has proven convenient attimes, principally for reasons of common usage, to refer to thesesignals as bits, values, elements, symbols, characters, terms, numbers,or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the followingdiscussions, it is appreciated that throughout the present invention,discussions utilizing terms such as “processing” or “accessing” or“executing” or “storing” or “rendering” or the like, refer to the actionand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories and other computer readable media into other data similarlyrepresented as physical quantities within the computer system memoriesor registers or other such information storage, transmission or displaydevices. When a component appears in several embodiments, the use of thesame reference numeral signifies that the component is the samecomponent as illustrated in the original embodiment.

This present invention provides a solution to the increasing challengesinherent in medical image automatic segmentation. In particular, variousembodiments of the present disclosure provide automatic segmentation ofa patient's two-dimensional medical images through the use of aknowledge base comprising a plurality of reference image sets, whereineach reference image set comprises a plurality of two-dimensionalslices. Each slice in a reference image set has been segmented andcontoured, with each structure of interest (e.g., organs, tumors,diseased tissues, etc.) labeled. As discussed in detail below, aplurality of reference image sets with similarities (e.g., similarsparse landmark signatures) to the medical image set are selected toform a base set and each deformed with a deformable image registrationalgorithm to generate mappings from the selected reference images to themedical image. Segmented structures of interest in each selectedreference image are also deformed to the medical image to generatemappings to the medical image. A weighted average segmentation for eachorgan of interest is calculated, wherein the weighted averagesegmentation for each organ of interest is used as a “seed” forstructure-specific automatic segmentation algorithms. After theautomatic segmentation algorithms for each structure of interest arecomplete, the results are reviewed and contours are corrected asrequired. After approving the medical image, the medical image and itsassociated meta-data, sparse landmark signatures and structures ofinterest are added to the knowledge-base.

Medical Image Segmentation

As is well known in the art, image segmentation is a process forreviewing a digitized medical image, and based on the merits of themedical image, differentiating or identifying structures of interest andlabeling them. For example, when reviewing a medical image of thethorax, using image segmentation, a line can be drawn around a sectionof the image and labeled as “lung.” In other words, everything withinthe line is deemed lung, and everything not inside the line is deemednot lung. Image segmentation is utilized in radiation therapies, where apractitioner needs to precisely define where the radiation treatment isto be placed and just as importantly, where the radiation treatment isnot to be placed. A structure of interest may be defined as: an organ; atumor; diseased tissues; and bones visible in the medical image, forexample. CT and MM scans, as mentioned above, are common medical imagemodalities. However, any two-dimensional image may be segmented where apractitioner is interested in differentiating between structures ofinterest in the image. In CT and MM scans where a plurality of slices isgenerated, segmentation is usually performed on each slice in turn. Forthe sake of convenience and simplicity, segmenting a CT scan, forexample, refers to segmenting each slice in turn.

Medical image segmentation is used with, for example, IMRT (intenselymodulated radiation), to paint radiation, or shape the radiation in thebody to conform to the tumor. Segmentation is a core benefit: when aradiation treatment may be shaped so as to conform to a tumor, criticalorgans can then be avoided. Rather than merely fitting or shaping theradiation therapy to fit the tumor, critical organs, such as the brainstem, for example, can be avoided. An exemplary radiation therapyutilizes pencil beams of radiation, placing radiation into the body attargeted locations. However, to do that, the computer running the IMRTtreatment needs to know where the brain stem is. Treatment planningsoftware for the IMRT takes medical images (e.g., CT or MM data sets)and identifies all the organs or structures of interest at risk(especially those structures that are sensitive to radiation, or thatmust not be irradiated regardless of their radiation sensitivity).Further, those structures of interest that are targets for the radiationtherapy are also identified. The structures as drawn on the individualslices are combined into volumes of interest, spanning multiple slices.Given the volumes drawn and the doses prescribed for each volumeidentified, the treatment planning software will calculate the bestmethod for putting the dose in.

But as medical imaging resolution increases providing additional slices,and the number of structures that must be identified, as well as theprecision with which they must be identified increases, the treatmentplanning process takes longer. This is especially apparent when eachslice must be segmented by hand. However, even with automaticsegmentation algorithms currently available, the practitioner is stilloften required to manually drop the seeds into the image, and mustreview each slice individually. “Dropping a seed” is a process whereby apractitioner selects points within an area of a medical image andidentifies them as part of a particular structure of interest (e.g., aprostate, bladder, or femur). Therefore, an excessive amount of time canbe spent manually seeding each medical image. However, even afterseeding the automatic segmentation algorithms, the practitioner is oftenrequired to edit the segmentation and contouring results to correctthem. This editing is required as the automated segmentation resultswill commonly produce some segmentation and/or contouring errorsrequiring practitioner editing to correct. The end result is that evenwith automatic segmentation software tools, the time required is oftenlittle better than manually segmenting; with the necessity to drop seedsand then frequently spend a significant amount of time editing tocorrect the autosegmentation and autocontouring software results.

Automatic segmentation is further complicated by the fact that automaticsegmentation tools are often tailored to be structure specific, suchthat they only work well on certain internal organs. For example, anautomatic segmentation tool utilizing edge detection has difficulty inautomatically individually segmenting the bladder and prostate, becausethey can't be individually differentiated. When the bladder is full andtouching the prostate, the prostate-bladder barrier can't be seen. Theend result is difficulty in determining where one ends and the other onebegins. Instead, automatic segmenting tools for the prostate are forcedto rely on modeling and looking at the curvatures of the bladder andprostate. And of course, even when using a structure-specific automaticsegmentation tool, the individual seeds must still be placed within eachstructure of interest.

Knowledge-Based Image Automatic Segmentation

FIG. 1 illustrates a knowledge-base 100 in accordance with an embodimentof the present invention. A knowledge base 100 comprises a plurality ofreference image sets 102 a-102 n. Each reference image set 102 a-102 ncomprises two-dimensional slices of a three-dimensional volume. Eachreference image set 102 a-102 n further comprises meta-data, a sparselandmark signature corresponding to each reference image, and aplurality of segments and contours corresponding to structures ofinterest seen in each individual reference image slice.

In one exemplary embodiment a knowledge-base 100 initially comprises aplurality of expert cases. Each of these expert cases contains referenceimage sets 102 a-102 n with segmentations and contours that have beenindividually hand-drawn. Each structure of interest has been labeled.

In another embodiment of the knowledge-base 100, the initial pluralityof reference image sets 102 a-102 n would share common characteristics.For example, each reference image set 102 a-102 n in the knowledge-base100, would have a common characteristic in the meta-data, such as race,sex, diagnosis, treatment site, age, and size of patient, etc. Aknowledge-base 100 used predominantly in the treatment of Orientalpatients would be different from a knowledge-base 100 used predominantlyin the treatment of Europeans, for example. Similarly, a knowledge-base100 used in an exclusively female practice would only need to comprisereference data sets 102 a-102 n of female patients.

FIG. 1 further illustrates a medical image processing module 110. Themedical image processing module 110 receives an image set for a currentpatient called a medical image set 112. The medical image processingmodule 110 processes the medical image set 112 to reduce each imageslice to a sparse landmark signature. The end result is a sparselandmark signature set 114.

A sparse landmark signature based on a medical image slice will allow upto 90% of the information contained in a medical image set to beignored. Rather than running the algorithms pixel by pixel across eachslice of a medical image set 112, which would consume an excessiveamount of computational time and resources, a sparse landmark signaturereduces the individual slices to only a collection of points for furtherprocessing.

For example, a medical image set 112 of a male patient with prostatecancer would have a particular sparse landmark signature set 114. If thediagnosis is prostate cancer, the three-dimensional volume of themedical image set 112 is the middle of the body. Because the pelvicgirdle defines the location of the prostate, the individual slices arescanned from top to bottom, looking for the first instance of bone(image pixels of bright intensity). This is accomplished for both sidesof the pelvic girdle, allowing the location of the femoral heads to beidentified. Next, the point where the pelvic girdle ends is alsolocated. Triangulating from those three points almost always locates theprostate. Therefore these three points are included as part of a sparselandmark signature set 114 of the medical image set 112. Lining up thepoints creates the signature. An arrangement of points allowspoint-based segmentation. After the points are emplaced the morecomplicated medical image slice isn't needed any more. In one exemplaryembodiment, once all the points are selected, an automatic segmentationtool library is accessed to select a drawing or model of the pelvicgirdle from the above example and stretch it until all the points arelined up. Whatever shape the contours have, it is now considered thestarting point for further refinement.

However, care must still be taken so as not to fool an automaticsegmentation tool. With a minimal sparse landmark signature it ispossible for an algorithm to mistake head and neck signatures (spine andshoulders) for pelvic girdle. Therefore, in exemplary embodiments,additional features are included in the sparse landmark signature. Thetotal number of points in a signature will be counted. For example, if asignature contains twelve points, prostate cases can be ruled out.Rather than prostate, a signature with twelve points identifies a headand neck case. Therefore, based on a full review of a signature whichidentifies for example, twelve points, a head and neck library would besearched for a head and neck model rather than a pelvic girdle model.

As further illustrated in FIG. 1, after the sparse landmark signatureset 114 of the medical image set 112 is generated, reference image sets102 a-102 n can be selected based upon their similarities to the presentpatient medical image set 112 and the corresponding sparse landmarksignature set 114. In an exemplary embodiment of the present invention,a plurality of reference image sets 102 a-102 n is selected. Exemplaryembodiments can select from as few as three reference image sets 102a-102 n to as many as ten or more reference image sets 102 a-102 n. Inone exemplary embodiment, three reference image sets 102 a-102 c areselected. The selected reference image sets 102 a-102 c become a baseset 116.

In an exemplary embodiment of the present invention, the medical imageprocessing module 110 begins the reference image set 102 a-102 nselection process by reviewing meta-data attached to the current medicalimage set 112. As is well known in the art, meta-data may includeinformation associated with a particular patient at a very high levelthat is not especially personalized. It would include such details as:age, sex, diagnosis, weight, whether or not there is nodal involvement,any co-morbidities, treatment site, side of the body, and approximatelocation of tumor. The available meta-data attached to the currentmedical image set 112 is used to start a search through theknowledge-base, reviewing the meta-data attached to reference image sets102 a-102 n for corresponding matches. Through the use of meta-data, theknowledge-base can be sorted so that in an exemplary medical image setwith meta-data identifying a male patient with advanced lung cancer,only reference image sets of males with lung tumors on the same side ofthe body are reviewed. From the knowledge-base, the top reference imageset 102 a-102 c matches are selected based upon their similarities tothe present medical image set 112 meta-data and sparse landmarksignature.

A base set 116 contains as noted above in exemplary embodiments,anywhere from three to ten or more reference image sets 102 a-102 n. Aplurality of reference image sets 102 a-102 n are selected to reduce theerror inherent in each individual reference image set 102 a-102 n. It isobvious that a single reference image set 102 a-102 n won't be a perfectmatch; there will be some disagreement between the reference image set102 a-102 n and the medical image set 112. A plurality of referenceimage sets 102 a-102 n in the base set 116 allows the formation of astatistical model to serve as a reference image.

In an exemplary embodiment of the present invention, each of theselected reference images sets 102 a-102 c in the base set (in thisexemplary embodiment, three reference cases are selected) isindividually deformed to the medical image set. In other words, thepoints in the sparse landmark signature of each reference image set 102a-102 c (one slice at a time) are deformed to the points in the sparselandmark signature set 114 of the medical image set 112. In oneexemplary embodiment, a deformable image registration algorithm willtake the selected reference image sets 102 a-102 c and the medical imageset 112 which has no segmentation or contouring and morph each of thereference image sets 102 a-102 c one at a time, and one slice at a time,to look like the medical image set 112. The registration algorithm willalso transfer the contours from each reference image set 102 a-102 c tothe medical image set 112 as well. The contours are each transferredalong with the labels. By deforming each of the three selected referenceimage sets 102 a-102 c to the medical image set 112, the contours thatwere tagged to a pixel will also move as the pixels move. Each of thepoints from each of the reference image sets 102 a-102 c are deformed tothe points of the medical image set 112. In another embodiment of thepresent invention, a portion of the selected reference image sets 102a-102 n in the base set are deformed to the medical image set. Thisportion can be one or more of the reference image sets 102 a-102 n.

As multiple reference image sets 102 a-102 c are individually deformedto the medical image set 112, exemplary embodiments of the presentinvention are therefore using a statistical-based algorithm, instead ofonly an image-based algorithm. By taking the results coming from thedeformation of the three reference image sets 102 a-102 n to the medicalimage set 112 and averaging them, statistical processing will provide afinal result that is an average of the three (or whatever number ischosen) reference images 102 a-102 c. It's more statistically likelythat the average for the three top references will converge on a morecorrect answer than that any one of them will be individually correct.As illustrated in FIG. 1, the medical image processing module willtherefore deform the base set 116 to the sparse landmark signature 114of the medical image set 112 to create a statistical model 118containing segmentation and contouring of the structures of interest inthe medical image set 112.

In an exemplary embodiment, weighted-average segmentation is therebyprovided to create the statistical model 118. If all three referenceimage sets 102 a-102 c agree very well, then a simple average of thethree will be sufficient. But if two of them agree closely and the thirdhad an outlier (e.g., a contour or segmentation result for a particularstructure of interest is different from the other results), the thirdresult's outlier could be de-emphasized in the statistical model 118,making the assumption that the outlier is incorrect. Weighted-averagesegmentation for example, would look at the standard deviation of theresults for each of the structures of interest that are identified inthe statistical model 118. A simple method of weighting the results whenan outlier appears would be to discard the outlier result. It should benoted that the statistical weighted-averaging is not applied to thepatient, i.e., medical image set 112, but to each of the resultingstructures of interest within the medical image set 112. In other words,in an exemplary embodiment there can be as many weighted mixes betweenthe three reference image sets 102 a-102 c as there are structures ofinterest. The statistical model 118 therefore provides a good startingpoint for further segmentation processing. The statistical model canwith a certain amount of statistical certainty, identify what isprostate or lung for example, even if the outline contours aren'tperfect. That is, at least what is inside the contour has beenidentified as prostate or lung, as the case may be.

In an exemplary embodiment, when the weighted-average segmentationprovided to the statistical model 118 is not approved by thepractitioner, additional reference image sets that likewise containcommon meta-data characteristics and similar sparse landmark signatureswill be selected and added to the base set 116 to be registered (i.e.,deformed) to the medical image set 112. In an exemplary embodiment, theadditional reference image sets are added one at a time. Again, asdiscussed above, the sparse landmark signature of the newly addedreference image set would also be deformed to the sparse landmarksignature of the medical image set as the original three reference imagesets were. After statistical processing, an updated weighted-averagesegmentation is provided to the statistical model 118 for review by thepractitioner. Additional reference image sets can be added to the baseset 116 as required, as long as additional candidate reference imagesets remain in the knowledge-base 100, and their addition to the baseset 116 continues to improve the weighted-average segmentation of thestatistical model 118.

The selected reference image sets 102 a-102 n also contain correspondingtreatment prescriptions, strategies, and IMRT dose objectives. By alsoproviding the treatment prescription, strategy and IMRT dose objectivesfor each of the selected reference image sets, they can provide areasonable starting point for the current patient. Further, statisticalparameters of the statistical model 118 may also be evaluated, bylooking at non-image based components from the reference image sets(shape, volume, smoothness, density, etc.) and thereby refine theresulting statistical model 118 and updated statistical model 120.

Knowledge-Based Image Sub-Volume Segmentation

In an exemplary embodiment, the process described above for a medicalimage set 112 and a base set 116 comprising a plurality of referenceimage sets 102 a-102 n is repeated for a plurality of structures ofinterest or “sub-volumes” of the medical image set from a plurality ofpatient cases in the knowledgebase, as illustrated in FIG. 4. Asub-volume will typically contain only a single structure of interestrather than all of the structures of interest found in the medical imageset. A plurality of reference image set sub-volumes 402 a-402 n, groupedas sub-volume base sets, for each identified structure of interest 412a-412 n of the medical image set 112 is selected. In other words, aplurality of reference image set sub-volumes 402 a-402 n are selectedfor each sub-volume (i.e., structure of interest) of the medical imageset. For example, for a medical image set sub-volume for the prostate, aplurality of prostate sub-volumes from reference image sets will beselected. In other words, rather than entire reference image sets 102a-102 n, only a prostate sub-volume 412 a-412 n of the reference imagesets 102 a-102 n would be selected. Again, in an exemplary embodiment,the number is three. Each plurality of reference image set sub-volumesis therefore a sub-volume base set for a particular structure ofinterest. There can be a sub-volume base set for each structure ofinterest 412 in the medical image set 112.

As described above for reference image sets 102 a-102 n and the medicalimage set 112, in the same manner, each reference image set sub-volume402 is deformed to a corresponding structure of interest 412 of themedical image set 112, just as each reference image set 102 is deformedto the medical image set 112. Further, as described above, in anotherembodiment of the present invention, a portion of the selected referenceimage sets sub-volumes 402 a-402 n in the plurality of sub-volume basesets is deformed to the structures of interest 412 a-412 n in themedical image set 112. This portion can again be one or more of thereference image set sub-volumes 402 a-402 n. After a weighted-averagesegmentation for each structure of interest in the medical image set 112is completed as described above, each structure of interest orsub-volume 412 a-412 n would also have a corresponding sparse landmarksignature set that is also a portion of the sparse landmark signatureset 114 of the medical image set 112. By selecting particular structuresof interest 402 (i.e., the best candidates) of reference image sets 102(and their corresponding sparse landmark signatures for the structuresof interest) to deform to the corresponding structure of interest 412 inthe medical image set 112 (and its corresponding sparse landmarksignature), the resulting weighted-average segmentation of eachstructure of interest 412 a-412 n used to create the statistical model118 can be further improved. This is because rather than seeking for areference image set 102 candidate that is a good all-around match to thepatient anatomy, only the individual sub-volumes 402 or structures ofinterest are considered. Therefore, a reference image set 102 that wouldnot be a good candidate for all structures of interest can still be agood sub-volume candidate.

Continuous Knowledge-Base Improvement

However, to get better results with the knowledge-base 100, additionalreference image sets need to be added to the knowledge-base 100. Inother words, additional clinical cases with their patient specificmeta-data, medical image sets, sparse landmark signatures, statisticalmodel can be added to the knowledge base after the practitioner hascompleted the patient-specific treatment plan and approved thesegmentation and contouring of the structures of interest in the medicalimage set. After the practitioner has reviewed, edited (as needed), andapproved the segmentation results, the subject patient's information canbecome a new reference image set with its own attached meta-data andtreatment information to be placed in the knowledge-base to increase thenumber of candidate reference image sets available. This new referenceimage set will then be available to be reused as part of a base set thenext time there is a patient with similar meta-data, etc. Even with verypersonalized, patient-specific medicine, these described processes allowthe creation of a closed loop: each iteration of the process allows theaddition of a new reference image set 102 to the knowledge-base.Therefore, as more reference image sets 102 are added to theknowledge-base 100, the more robust the knowledge-base 100 becomes, withimproved statistical modeling and better end results. As more referenceimage sets 102 are placed into the knowledge-base 100, the searches canbe further refined and based on additional criteria to ensure that theselected reference image sets 102 a-102 n are close matches to themedical image set 112.

Statistical Model Used as a Seed

In an exemplary embodiment, the statistical model 118 with itsweighted-average segmentation of each structure of interest can be usedas a seed itself. Rather than traditional structure-specificsegmentation tools requiring manually selected seeds before theprocessing begins, the seeds can be provided by the identifiedstructures of interest from the statistical model 118. By providingseeds that are statistically probable to be drawn in pixels inside thestructure of interest, it allows any of the previously discussedsegmentation methods to begin with improved seed selections. Further,using the statistical model's weighted-average segmentation ofstructures of interest allows these other segmentation methods to avoidhaving to start from scratch.

While there is no guarantee that a structure of interest will besegmented correctly, there is a good statistical possibility that acluster of pixels in the middle of the segmentation of the structure ofinterest will be within that actual structure of interest. In anexemplary embodiment, this central cluster of statistically significantpixels is then used as the “seed” for other automatic segmentationalgorithms.

In an exemplary embodiment, as illustrated in FIG. 1, and described inmore detail below, the statistical model 118 is used as a seed fordeformable image registration and to drive the autosegmentationcomponent of structure-specific automatic segmentation tools 130 a-130n. In an exemplary embodiment, the automatic segmentation tools 130a-130 n can be either structure-specific segmentation tools ornon-structure-specific segmentation tools. When the segmentation resultsare returned by the automatic segmentation tools 130 a-130 n to themedical image processing module 110 the results can be used to create anupdated statistical model 120 that comprises weighted-averagesegmentation of each structure of interest in the medical image set 112,as discussed in detail below.

The automatic segmentation tools 130 a-130 n use the seed provided bythe statistical model 118 to calculate the segmentation of one or morestructures of interest. These structure-specific segmentation algorithmscan use a structure-specific model to aid in completing the segmentationthat will look at all the data within the image that has been labeled asa seed for the structure of interest (e.g., the prostate or lung). Thisgives the structure-specific algorithm a very good starting point. Thestructure-specific algorithms work best when they are working on thespecific structures they are designed to segment.

The statistical weighted averaging can also be extended to statisticallyweighted-averaging of these different algorithms depending on theirabilities and whether there is something about the particular meta-dataor the image that would make one algorithm more accurate than another.This allows the results from a plurality of algorithms to be averaged toarrive at an even more statistically accurate definition of a structureof interest by comparing the results from a plurality ofstructure-specific algorithms. The end result would be the medical imageprocessing module calculating an updated statistical model 120 thatcomprises a weighted-average segmentation of each structure of interestin the medical image set 112. As discussed above, this updatedstatistical model 120 after being reviewed, editing, and finallyapproved by the practitioner for use with the current patient, may beadded to the knowledge-base as a new reference image set 102.

Knowledge-Based Autosegmentation Algorithm Improvements

In an exemplary embodiment, the above described processes can include anupdate to the statistical tools, by performing a difference calculation.The difference calculation compares what the system calculated to whatwas ultimately accepted by the practitioner, identifies the point(s)that failed, and acts to modify the behavior of the algorithm. Byback-projecting the difference, changes are made to the algorithmconfiguration until an optimal solution results. Weighting functions canbe adjusted to reach the optimal results.

In other words, if the system was failing to recognize a pelvis case,for instance, the selection of sparse landmarks would have to beadjusted so that a pelvis case is properly recognized. In anotherexample, the way that deformal image registration is identifying astructure of interest may have to be changed if the location is correct,but the segmentation/contour result is incorrect. For example, if a gapexists between the computed and hand-drawn edge of an organ of interest,then the edge detection algorithm can be adjusted to correct it. Knowingwhere the gap was and where the correction was made, the correction canbe worked into the particular algorithm to improve it.

Knowledge-Based Autosegmentation Processes

FIG. 2 illustrates the steps of a process in accordance with anembodiment of the present invention. Where the process includes stepsthat have already been explained in detail, they will not be repeatedhere. In step 202, a new medical image set 112 is loaded. The medicalimage set 112 may be loaded into the memory of a computer system. Instep 204, landmark features (i.e., points on the medical image set 112)are extracted to create a sparse landmark signature 114.

In step 206, a knowledge-base 100 is scanned to search for referenceimage sets 102 a-102 n with similar sparse landmark signatures 114 andsimilar meta-data. In an exemplary embodiment, the sparse landmarksignatures 114 serve as an index for the knowledge-base 100. In step208, n reference image sets 102 a-102 n are selected to form a base set116. In an exemplary embodiment, n=3. In step 210, a portion of thereference image sets 102 a-102 n is registered to the medical image set112. This portion can be two or more of the selected reference imagesets 102 a-102 n. In one exemplary embodiment, all of the selectedreference image sets 102 a-102 n are registered. By deforming thereference image sets to the medical image, the segmented and contouredstructures of interest in the reference image sets are also applied tothe medical image set.

In step 212, a weighted average structure-specific segmentation iscompleted to form the statistical model 118. In other words, weightedaverages are calculated for each structure to reach a weighted-averagestructure segmentation. As noted above, the statistical model 118 maynow serve as a seed for use in other segmentation tools, including thosethat are structure-specific.

In step 214, the practitioner or user reviews, edits and approves thesegmentation results. In step 216, structure-specific statisticalparameters are computed and improved when possible (by looking atnon-image based components from the reference image sets (shape, volume,smoothness, density, etc.). Then in step 218, the knowledge-base isupdated with the medical image set 112 and its corresponding structureset (statistical model 118), sparse landmark signature set 114, andmeta-data as a new reference image set 102.

FIG. 3 illustrates the steps of a process in accordance with anembodiment of the present invention. Where the process includes stepsthat have already been explained in detail, they will not be repeatedhere. Step 302 begins after a statistical model 118 has been calculated.In step 302, statistical parameters describing each segmented structureof interest are further evaluated. In step 304, sub-volumes 412 a-412 nfrom the medical image set 112, wherein a sub-volume contains or is astructure of interest, are loaded into the computer for processing.These sub-volumes 412 a-412 n are the structures of interest with theirown sparse landmark signatures that have already been segmented in FIG.2, as described above. In other words, each sub-volume 412 a-412 ncomprises a sub-volume sparse landmark signature. In step 306, nreference image set sub-volumes 402 a-402 n are loaded for eachstructure of interest (i.e., sub-volume) in the medical image set 112 toform corresponding sub-volume base sets. In one embodiment, n=3. Each ofthe reference image set sub-volumes 402 a-402 c of the plurality ofsub-volume base sets is selected for a corresponding structure ofinterest 412 a-412 n in the medical image set 112. In one embodiment,each reference image set sub-volume 402 can come from a differentreference image set 102. In other words, if there are fifteen structuresof interest in the medical image set 112, then there will be forty-fivereference image sets 102 a-102 n providing the necessary forty-fivereference image set sub-volumes 402 a-402 n contained in the fifteensub-volume base sets. In another embodiment, a single reference set 102can provide a plurality of reference image set sub-volumes 402 a-402 n.

In step 308, a portion of the reference image set sub-volumes 402 a-402n and their segmented and contoured structures of interest are deformedor registered to the corresponding structure of interest 412 a-412 n ofthe medical image set 112. This portion can be two or more of theselected reference image sets 402 a-402 n. In one exemplary embodiment,all of the selected reference image set sub-volumes 402 a-402 n areregistered or deformed, with the reference image set sub-volumes 402a-402 n from each sub-volume base set deforming to the correspondingstructure of interest 412 a-412 n of the medical image set 112. In step310, a weighted-average segmentation value is calculated for eachstructure of interest 412 a-412 n in the medical image set 112.Calculating the weighted-average segmentation value for each structureof interest 412 a-412 n in the medical image set 112 allows thecalculation of an updated statistical model 118.

In step 312 each structure of interest 412 a-412 n in the statisticalmodel 118 is used as a seed and each post-processed by a plurality ofstructure-specific segmentation tools. Each structure of interestsegmentation is used as a seed for a plurality of structure-specificsegmentation tools. In step 314, a weighted-average segmentation resultis calculated for each structure of interest based upon the segmentationresults for each structure of interest 412 a-412 n provided by thestructure-specific segmentation tools. As mentioned above, thisweighted-average segmentation result for each structure of interest 412a-412 n is the updated statistical model 120.

In step 316, the practitioner or user reviews, edits and approves theweighted-average segmentation results from step 314. In step 318,structure-specific statistical parameters are computed and improved whenpossible (by looking at non-image based components from the referenceimage sets (shape, volume, smoothness, density, etc.). Then in step 320,the knowledge-base is updated with the medical image set 112 and itscorresponding structure set (statistical model 120), sparse landmarksignature 114, and meta-data as a new reference image set 102.

FIG. 5 illustrates the steps of a process in accordance with anembodiment of the present invention. In step 502, a cohort of patientsor reference image sets 102 a-102 n within the knowledge-base areselected that share common characteristics. These common characteristicscan include diagnosis, sex, age, race, treatment site, etc. In step 504,all members of the cohort are deformed into a single aggregated dataset,forming a single representative case. In step 506, geometric analysis isperformed on the radiation therapy for structures of interest and theassociated radiation therapy doses. In step 508, cohort representativesare compared to other cohort representative cases.

Although certain preferred embodiments and methods have been disclosedherein, it will be apparent from the foregoing disclosure to thoseskilled in the art that variations and modifications of such embodimentsand methods may be made without departing from the spirit and scope ofthe invention. It is intended that the invention shall be limited onlyto the extent required by the appended claims and the rules andprinciples of applicable law.

What is claimed is:
 1. A method for segmenting new image data sets, themethod comprising: receiving a medical image set that includes astructure of interest of a plurality of structures of interest, whereinthe medical image set is for a particular patient; searching aknowledge-base for representative matches to the medical image set toform a base set, wherein the base set comprises a plurality of referenceimage sets; deforming at least a portion of the plurality of referenceimage sets of the base set to generate mappings from the base set to themedical image set and to create a statistical model comprisingsegmentation and contouring of the structure of interest in the medicalimage set; using the statistical model to provide a seed that identifiespoints in the medical image set that are part of the structure ofinterest, wherein the seed comprises a cluster of pixels around themiddle of the segmentation of the structure of interest in the medicalimage set; determining a weighted-average segmentation for the structureof interest using the seed provided by the statistical model; andupdating the statistical model with results from the weighted-averagesegmentation for the structure of interest.
 2. The method of claim 1,wherein the base set is formed from a cohort of reference image setswithin the knowledge-base that share common characteristics, wherein thecommon characteristics comprise at least one of: sex, diagnosis, stage,responder versus non-responders, age, treatment site, and size.
 3. Themethod of claim 1, further comprising: computing a sparse landmarksignature based on the medical image set, wherein the sparse landmarksignature comprises a number of points, wherein the structure ofinterest in the medical image set is identified based on the number ofpoints in the landmark signature; and using the sparse landmarksignature to select the reference image sets included in the base set.4. The method of claim 1, wherein deforming at least a portion of theplurality of reference image sets of the base set, comprises: deformingthe at least a portion of the plurality of reference image sets of thebase image with a deformable image registration algorithm.
 5. The methodof claim 1, wherein deforming at least a portion of the plurality ofreference image sets of the base set, comprises: deforming segmentedstructures of interest from the at least a portion of the plurality ofreference image sets of the base set to create a map to the medicalimage set; deforming points of a plurality of sparse landmark signaturesof the at least a portion of the plurality of reference image sets ofthe base set to the points of the sparse landmark signature of themedical image set; comparing meta-data from the medical image set tometa-data of the reference image sets in the knowledge-base, andcomparing sparse landmark signatures of the medical image set to sparselandmark signatures of the reference image sets in the knowledge-base.6. The method of claim 1, further comprising: reviewing and correctingcontours from the weighted-average segmentation to produce an approvedmedical image set; and adding the approved medical image set andcorresponding structures of interest to the knowledge-base as a newreference image set.
 7. A method for segmenting new image data sets, themethod comprising: receiving a medical image set that includes astructure of interest of a plurality of structures of interest, whereinthe medical image set is for a particular patient; searching aknowledge-base for representative matches to the medical image set toform a base set, wherein the base set comprises a plurality of referenceimage sets; deforming at least a portion of the plurality of referenceimage sets of the base set to generate mappings from the base set to themedical image set and to create a statistical model comprisingsegmentation and contouring of the structure of interest in the medicalimage set; using the statistical model to provide a seed that identifiespoints in the medical image set that are part of the structure ofinterest, wherein the seed comprises a cluster of pixels around themiddle of the segmentation of the structure of interest in the medicalimage set; determining a first weighted-average segmentation for thestructure of interest using the seed provided by the statistical model;searching the knowledge-base for representative matches to the medicalimage set to form a plurality of sub-volume base sets, wherein asub-volume base set comprises a plurality of reference image setsub-volumes; deforming at least a portion of the plurality of referenceimage set sub-volumes of the plurality of sub-volume base sets togenerate mappings from the plurality of sub-volume base sets tocorresponding sub-volumes of the medical image set; determining a secondweighted-average segmentation for the sub-volumes in the medical imageset using segmentation results provided by structure-specificsegmentation tools for the sub-volumes in the medical image set; andupdating the statistical model with results from the secondweighted-average segmentation.
 8. The method of claim 7, wherein thebase set is formed from a cohort of reference image sets within theknowledge-base that share common characteristics, wherein the commoncharacteristics comprise at least one of: sex, diagnosis, stage,responder versus non-responders, age, treatment site, and size.
 9. Themethod of claim 7, further comprising: computing a sparse landmarksignature based on the medical image set, wherein the sparse landmarksignature comprises a number of points, wherein the structure ofinterest in the medical image set is identified based on the number ofpoints in the landmark signature, and wherein the sparse landmarksignature for the medical image set further comprises a plurality ofsparse landmark signatures for the sub-volumes in the medical image set.10. The method of claim 9, wherein deforming at least a portion of theplurality of reference image sets of the base set and deforming at leasta portion of the plurality of reference image set sub-volumes of theplurality of sub-volume base sets, comprises: deforming points of aplurality of sparse landmark signatures of the at least a portion of theplurality of reference image sets of the base set to the points in thesparse landmark signature of the medical image set; and deforming pointsof a plurality of sparse landmark signatures of the at least a portionof the plurality of reference image set sub-volumes of the plurality ofsub-volume base sets to the points in the corresponding sparse landmarksignatures of the plurality of structures of interest of the medicalimage set.
 11. The method of claim 9, wherein searching theknowledge-base for representative matches to form a base set, comprises:comparing meta-data from the medical image set to meta-data of referenceimage sets in the knowledge-base, comparing sparse landmark signaturesof the medical image set to sparse landmark signatures of referenceimage sets in the knowledge-base; and comparing sparse landmarksignatures of each structure of interest of the medical image set tosparse landmark signatures of corresponding structures of interest ofreference image sets in the knowledge-base.
 12. The method of claim 7,wherein deforming at least a portion of the plurality of reference imagesets of the base set and deforming at least a portion of the pluralityof reference image set sub-volumes of the plurality of sub-volume basesets, comprises: deforming the at least a portion of the plurality ofreference image sets of the base image with a deformable imageregistration algorithm; and deforming the at least a portion of theplurality of reference image set sub-volumes of the plurality ofsub-volume base sets with the deformable image registration algorithm.13. The method of claim 7, wherein deforming at least a portion of theplurality of reference image sets of the base set and deforming at leasta portion of the plurality of reference image set sub-volumes of theplurality of sub-volume base sets, comprises: deforming segmentedstructures of interest from the at least a portion of the plurality ofreference image sets of the base set to create a map to the medicalimage set; and deforming segmented structures of interest from the atleast a portion of the plurality of reference image set sub-volumes ofthe plurality of sub-volume base sets to create a map to thecorresponding structures of interest of the medical image set.
 14. Themethod of claim 7, further comprising: reviewing and correcting contoursfrom the second weighted-average segmentation to produce an approvedmedical image set; and adding the approved medical image set andcorresponding structures of interest to the knowledge-base as a newreference image set.
 15. A non-transitory computer readable storagemedia having computer-readable and computer-executable program codeembodied therein for causing a computer system to execute a method ofsegmenting new image data sets, the method comprising: receiving amedical image set that includes a structure of interest of the pluralityof structures of interest; accessing a statistical model comprisingweighted-average segmentations of a plurality of structures of interestincluding a segmentation of the structure of interest; using thestatistical model to provide a seed that identifies points in themedical image set that are part of the structure of interest, whereinthe seed comprises a cluster of pixels around the middle of thesegmentation of the structure of interest in the medical image set;computing a sparse landmark signature based on the medical image set,wherein the sparse landmark signature comprises a number of points,wherein the structure of interest in the medical image set is identifiedbased on the number of points in the landmark signature; searching aknowledge-base for representative matches to form a base set, whereinthe base set comprises a plurality of reference image sets, wherein thesparse landmark signature is used to select the reference image setsincluded in the base set, and wherein the base set is formed from acohort of reference image sets within the knowledge-base that sharecommon characteristics, wherein the common characteristics comprise atleast one of: sex, diagnosis, stage, responder versus non-responders,age, treatment site, and size; deforming a portion of the plurality ofreference image sets of the base set to generate mappings from the baseset to the medical image set; determining a weighted-averagesegmentation for a plurality of structures of interest in the medicalimage set; using the weighted-average segmentation for the structure ofinterest as a seed for at least one structure-specific segmentationalgorithm; updating the statistical model with results from theweighted-average segmentation; reviewing and correcting contours fromthe weighted-average segmentation to produce an approved medical imageset; and adding the approved medical image set and correspondingstructures of interest to the knowledge-base as a new reference imageset.
 16. The non-transitory computer readable storage media of claim 15,wherein said deforming a portion of the plurality of reference imagesets of the base set, comprises an operation selected from the groupconsisting of: deforming the portion of the plurality of reference imagesets of the base image with a deformable image registration algorithm;deforming segmented structures of interest from the portion of theplurality of reference image sets of the base set to create a map to themedical image set; and deforming points of a plurality of sparselandmark signatures of the portion of the plurality of reference imagesets of the base set to the points of the sparse landmark signature ofthe medical image set; and wherein said searching the knowledge-base forrepresentative matches to form a base set, comprises: comparingmeta-data from the medical image set to meta-data of the reference imagesets in the knowledge-base, and comparing sparse landmark signatures ofthe medical image set to sparse landmark signatures of the referenceimage sets in the knowledge-base.